Repository navigation
Add data_quality_monitors bundle resource (direct engine) - #7094
Open
kevinlian2it wants to merge 10 commits into
Open
kevinlian2it wants to merge 10 commits into
kevinlian2it wants to merge 10 commits into
Conversation
Adds the data-quality/v1 Monitor API as a direct-engine bundle resource, alongside the legacy quality_monitors resource it supersedes. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
Integration test reportCommit: dcd7928
Top 6 slowest tests (at least 2 minutes):
|
Comment on lines
+15
to
+21
| ignore_remote_changes: | ||
| # Create-time options for the built-in dashboard; quality_monitors treats the | ||
| # same fields as not returned on read. | ||
| - field: data_profiling_config.skip_builtin_dashboard | ||
| reason: input_only | ||
| - field: data_profiling_config.warehouse_id | ||
| reason: input_only |
Member
There was a problem hiding this comment.
why can't these be set on the protos? then they would become generated with spec:input_only?
Author
There was a problem hiding this comment.
Good point: neither field is returned on read, so they should be INPUT_ONLY in the proto. I'll send the proto change. Until it reaches the SDK, the hand-written entries are needed; I'll remove them in the SDK bump that picks up the generated ones.
| // dataQualityMonitorUpdateMask lists every client-settable field; object_type | ||
| // and object_id are the identity and recreate on change. This matches the mask | ||
| // the Terraform provider sends. | ||
| const dataQualityMonitorUpdateMask = "anomaly_detection_config,data_profiling_config" |
Member
There was a problem hiding this comment.
DABs uses update_mask: * - does the API support this?
Author
There was a problem hiding this comment.
Yes. The v1 update goes through ApiUtil.update, which accepts * (Monitor has no IMMUTABLE fields), and we've exercised a * mask against staging. Switched to "*".
One caveat since SDK v0.191.0: anomaly_detection_config.notification_settings (the schema alert) is replaced on any * update, so omitting it removes an alert set in the UI. I'm treating the bundle as the source of truth, same as other resources, and documented it on the resource (22f23f8).
Comment on lines
+207
to
+210
| s := m.DataProfilingConfig.Schedule | ||
| if s != nil && s.PauseStatus != dataquality.CronSchedulePauseStatusCronSchedulePauseStatusUnpaused { | ||
| m.DataProfilingConfig.Schedule = nil | ||
| } |
Member
There was a problem hiding this comment.
would be good to cover this with an acceptance test
Author
There was a problem hiding this comment.
Added acceptance/bundle/data_quality_monitor, which validates a dev-mode target (schedule dropped) and a production target (schedule kept).
…ceptance test Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 14 changed tests (limit=20, 0 not selected) 660 tests: regression=16 unreleased=0 coverage=644 failing=0
16 more failure details omitted, see the full report in the run. |
…y monitors Co-authored-by: Isaac <no-reply@databricks.com>
…n alert Co-authored-by: Isaac <no-reply@databricks.com>
kevinlian2it
marked this pull request as ready for review
October 29, 2026 18:23
kevinlian2it
requested review from
janniklasrose and
lennartkats-db
and removed request for
lennartkats-db
October 29, 2026 18:23
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Adds a
data_quality_monitorsbundle resource for the data-quality/v1 Monitor API (anomaly detection on schemas, data profiling on tables). Direct engine only. The legacyquality_monitorsresource is unchanged; deprecating it is a separate follow-up.bundle/config/resources/data_quality_monitor.goembedsdataquality.Monitor. The ID is<object_type>,<object_id>, the same as the Terraform import ID fordatabricks_data_quality_monitor.bundle/direct/dresources/data_quality_monitor.go: CRUD onw.DataQuality. Update sendsupdate_mask: *.configs/data_quality_monitors.yml:object_typeandobject_idare ID fields;data_profiling_config.assets_dirrecreates on change (the update API ignores it). Output-only fields and the input-onlyskip_builtin_dashboard/warehouse_id(never returned on GET) come from the generated config.quality_monitors:run_asis rejected (the API has none), and presets that pause triggers dropdata_profiling_config.schedule(itspause_statusis read-only). The monitor has no name, so there is no name prefix. It has no permissions and no workspace URL, becauseobject_idis a UUID.Why
Data quality monitoring is moving to GA on the data-quality/v1 API, and
quality_monitorsonly covers the legacy table-monitor API. This resource lets bundles manage v1 monitors, including schema-level anomaly detection, which the legacy resource can't express.Notes for reviewers:
ListMonitoris unimplemented server-side, so lookup variables aren't supported for this resource.snapshot: {}is lost when a monitor is defined in Python.quality_monitorshas the same pre-existing limitation; it's visible in thedata_quality_monitors-supportgolden.*,anomaly_detection_config.notification_settingsis replaced on every update, so omitting it removes an alert set in the Data Quality Monitoring UI. The bundle is the source of truth; this is documented on the resource.data_quality_monitorsto the deployment metadata service, and databricks-eng/universe#2867273 marked the input-only fields in the spec (in SDK v0.191.0).Tests
all_test.go,state_load_test.go,resources_test.go, and the mutator tests.apply_target_mode_testasserts the dev-mode schedule drop.resources/data_quality_monitors/basic: schema monitor that references${resources.schemas.x.schema_id}. Covers create, in-place update, no drift, recreate on an identity change, and destroy.data_profiling: table monitor. Covers output-only fields not drifting, in-place schedule update, andassets_dirrecreate.remote-delete: an out-of-band delete is detected.lifecycle: cloud-capable.bundle/data_quality_monitor: dev-mode target drops the schedule; production keeps it.deployment/bind/data_quality_monitor: bind and unbind, and a malformed ID is rejected.python/data_quality_monitors-support.golangci-lint run ./...reports 0 issues. tidy, whitespace, deadcode, codeowners and lockfile checks pass, and all generators are idempotent. 505 PyDABs tests pass.lifecyclepasses on all clouds, with and without deployment history (DMS).This pull request and its description were written by Isaac.